What Looks Good with my Sofa: Multimodal Search Engine for Interior Design

摘要

In this paper, we propose a multi-modal search engine for interior designthat combines visual and textual queries. The goal of our engine is to retrieveinterior objects, e.g. furniture or wall clocks, that share visual andaesthetic similarities with the query. Our search engine allows the user totake a photo of a room and retrieve with a high recall a list of itemsidentical or visually similar to those present in the photo. Additionally, itallows to return other items that aesthetically and stylistically fit welltogether. To achieve this goal, our system blends the results obtained usingtextual and visual modalities. Thanks to this blending strategy, we increasethe average style similarity score of the retrieved items by 11%. Our work isimplemented as a Web-based application and it is planned to be opened to thepublic.